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ABSTRACT 

Particular probless in the process of sussative 
evaluation of docusent based inforsation systess designed for 
educators are caused by a series of conditions of conteit factors of 
the evaluation. Sose of these factors are identified and described. 
An evaluation plan is presented which recossends that the esphasis be 
shifted froB initial definitions of objectives, benefits, and clients 
to increased esphasis on the identification of alternative 
inforsation systess and the dissesination of its evaluation data* 
Exasples froB previous evaluation studies are presented to illustrate 
the probless as well as the recossended approach. fWCH) 
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A MODEL FOR EVALUATING DOCUMENT BASED 
EDUCATIONAL INFORMATION SYSTEMS 



ABSTRACT: Particular problems In the procatt of 
•unBBatlve evaluation of document bated Information 
systems designed for educators are caused by a series of 
conditions of context factors of the evaluation. Some 
of these factors are Identified and described • An evalu- 
ation plan is presented In vhlch a shift In emphasis on 
the importance of Initial definitions of objectives, 
benefits, and clients, and increased emphasis on tha 
Identification of alternate Information systems and the 
dissemination of evaluation data. Is recommended. Ex- 
amples from previous evaluation studies are presented to 
Illustrate cne problems and the recommended approach* 



The general themes of the URISA Conference for this year have been Identified 
In the Plenary Sessions as "Resources" and "Results"* The study of the results of 
an Information system, an activity that ultimately Is tied to considerations 
rolaU's^d to xesources for the establishment and maintenance of that system. Is part 
of the concerns of a comprehensive evaluation study* Such an evaluation st\idy of 
an infot'jation system wculd have to Include activities designed to obtain Informa- 
tion related to questions In three major categories* Questions In one category are 
concetp.«d with the operational characteristics of the system* The evaluation plan 
must Include provloions to Identify, describe, and clarify the functions of various 
component parts of the system and determine the costs associated with these compa- 
ny r.ts* The second major category consists of questions related to the Identifica- 
tion of the nature of the outcomes and objectives of the Information system and the 
determination of the degree to which these objectives have been met* Finally, the 
evaluation plan would noc be truly comprehensive unless Information was provided 
so that Judgments of the value of the objectives themselves, or the benefit of the 
Information system, could be made* 

Different groups have varying motives for seeking evaluation Information* 
System managers are usually most concerned with the first category while system 
supports must direct their attention to questions of objective attainment and the 
resulting benefit* The terms "formative" and "sunmatlve" evaluation are often used 
In the field of education to distinguish between roles which evaluation must play, 
guiding system Improvement and deciding about system acceptance* These roles often 
overlap and It Is Impossible to treat thrna as completely Independent but the concern 
for examining results of an Information system seems to dictate directing primary 
attention to sunmatlve evaluation* 



lUTRCDUCTION' 



This attention la neceaaary baeauae evan thoush thare may b« a trend, aa 
deacribed by Krevltt and Griffith, to focua on the effectlveneaa of Infotmatlon 
ayatema Inataad of on queatlona of ultlnate value It alao aeena quite true, aa 
these aame authora have Indicated, that the people who fund and aupport Infonnatlon 
ayatema are not going to be aatlafled without soma evidence of Impact or reaulta. 
Similar conmenta appear frequently In the literature* 



EXAMPLES OF PREVIOUS EVA.;DATI0MS 

In order to clarify the problema which occur In educational Information ayatem 
evaluations four examplea will be uaad throughout thla paper. All four are related 
In acme way to ERIC, Educational Reaourcea Information Canter, which la a national 
oyatem dealgned to provide educatora with acceaa to educational literature* ERIC 
provldea for monitoring, acquiring, evaluating, abatractlng, and indexing documenta 
repreaentlng current literature In education* In addition, ERIC apoa^ora Informa- 
tion analyala actlvltlea and publlcatlona and dlaaamlnatlon projecta dealgned to 
Increaae uae of current knowledge* The system is built on a decentralised design 
in which dearlnghouaea handle all document proceaaing actlvltlea* Tha entire ayatec 
la coordinated and Integrated In a central facility located within the National 
Inatitute of Education* In addition to acquiring and annoancing documenta, ERIC 
alao aupport a a document reproduction aervlce through which educatora csn obtain 
coplea of non-copyrighted materlala* The wide variety of typea of documenta in the 
ayatea Includea journal artlclea, reaearch reporta, conference proceedinga, profea- 
alonal papera. Innovative program descriptions, and bibliographies* 

With such an extensive Information aystem It would be ucusual to flue! &r.y sl^iglc 
overall evaluation atudy* Instead, a few atudlea which have ^ncentrated on certain 
parte of ERIC will be cited* One atudy, which examined ERIC producta and aervicea, 
waa deacribed in a report published in 1972 by Bernard Tty.^ The objectivea of the 
evaluation were relatively broad since there ware many types of producta examined* 
The major methoda uaad in the study were queationnar lea , aite vlaita, and consulta- 
tion with an adviaory board* A second study, conducted by Judith Wanger and reported 
in 1972, waa also concerned with the evaluation of producta but in thla atudy only 
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one type, Information analysis products, was examined*^ Questionnaires dlstrlbtttsil 
to both a random and a nonrandom sampla were used to determine ways the products 
are used, needs they meet, user satisfaction, and impact on educational practice* 
The third report differs from the preceding two In a number of ways* It is not a 
•tudf In the same sense since no direct data gathering activities are described* 
Also, It was not commissioned by ERIC sponsors* The third report Is an opinion 
paper written by William Paisley In which he examines ERIC and mikes suQgestlons 
for changes In the system*^ 

The fourth evaluation study describes the results of the txro year examination 
of the Pilot States Disseminata Ion Program*^ This program was set up In three 
states for a tryout period* The program had three major components* Two of these 
were an Information retrieval center and staff located in the State Education 
Agency and field agents assigned to work with* clients in designated target areas 
in order to assist in identifying problems* referring requests to the retrieval 
center, and delivering information back to the clients* Administration and manage- 
ment was the third component* The projects were relatively small with most states 
having two agents, three on the retrieval staff and one project director* The 
actual evaluation study was unusual in its concern not only for the evaluation but 
with uhe study of the evaluation process itself* The report contains much informatlua 
both in the text and in the appendices related to methodology of evaluation in general 
and foruiative evaluation in particular* Questionnaires, tape recorded case studlM, 
site visits, and shadow studies by field observers were some of the methods used 
in this inveatlgation* 



DIFFICULTIES IN SUMM(VTIVE EVALUATIONS 

Performing a sumoative evaluation of any information system is difficult* 
Problems arise from at least six factors in the evaluation context* In addition, 
when these six factors all interact, as they often do in evaluations of document 
based information systems designed for educators, the difficulties are greatly 
compounded* The general factors of: 

1* the nature of the information in the system 

2* the uses to which this information is put by clients of the system 
3* the nature of the organlsi^tional structure of the client's institution 
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A. the ability to link means and objectives of the client's Institution 
In causal chains 

S« the assumptions about decision models on which the evaluation plan 
Is formed 

6. the state of development of the Information system at the point of 
the summatlve evaluation 

all Impose special considerations for information system evaluation studies. 
Figure 1 presents a brief susmary of the six context factors along with a few 
of the possible conditions for each factor* 



Figure 1. Context Factors and Conditions In 
Information System Evaluations. 
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The broken line In Figure 1 connects the conditions which lead to the most 
severe problems In summatlve evaluations and the solid line connects conditions 
vhlch are less threatening. The evaluation of any Information system may be con- 
fronted with varying conditions of context factors but It Is not unreasonable, In 
ficC It Is even quite common, for an educational Information system evaluation to 
present the most severe case. The examples of evaluations of educational Informa- 
tion systems presented In this paper all represent, with only slight vcrlatlons on 
the last context factor, the right hand column of Figure 1. 



Nature of Information 

If an information system ia based on retrieval and dissemination of documents 
rather than specific data, the determination of the benefit of the system becomes 
difficult due to considerations of the quality of the document. Documents represent 
the results of a considerable amount of summation, organisation, and judgment by 
individuals other than the Information system clients. Various Investigators of 
tha quality of published educational research reports have estimated the percent of 



good reports to be between a low of eeven percent and a high o£ thirty percent. ' 
Even granting that the second figure Is correct, the proportion Is distressingly 
low. Information systems based on ERIC Indexes, a major source In all of the exam- 
ples, will face the problem of trying to determine whether any evidence of lack of 
benefit Is due to a failure of the Information system or Inadequacy or error In 
retrieved documents. A data based system, such as a managencent Information system, 
faces fewer sources of error In the transmitted Information. 

Information Use 

Closely related to the question of the nature of the Information In the system 
Is the consideration of the uses to which the Information will be put by the client. 
As an example of some of those uses Brlttaln, In a review of Information studies 
ir.vestlgatlng educational practitioners, lists direct Incorporation In lectures, 
developing research projects, serving as a stimulus for thought, or serving as a basis 
for some direct action such as goal setting or decision making.^ If information used 
for decision making can be related to a specific decision It will be possible to gain 
nn Idea of the benefit of the system being evaluated by seeking Information about 
the decision outcome. This Information can be used to make judgments of the benefits 
of the Information system In terms of the benefits of the decisions. However, when 
information Is used for Increasing awareness, it is extremely difficult to attach 
any direct measure of benefit* 

The extent to which the use of information for purposes other than actual deci- 
sion making occurs in educational settings is quite high. For example, in the Wanger 
evaluation of ERIC products, the following conclusion is stated. '*rhere is no clear 
evidence that information In products is used specifically in decision-making situa- 
tions."^ In the ?llot States evaluation a utilization index ^as developed in which 
one category represented the implementation of a specific practice or program 
resulting from efforts of the dissemination program* The average response rate in 
this category frou the three states vas eight percent. There was a higher response 
rate, approximately fourteen percent, for the category which represent use of inlor- 
tiAtion for planning vhich may have led to a rational decision of non-implementation, 
but this results was not clearly identified and it would be difficult to establish 
cin-r benefit estlrates frca this data. Fry also found that a large ERIC client group 
was graduate students who used retrieved information for research projects, assign- 
iti:ats, and term papers.^® In summary. It seems that a common use of Infomation by 
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educators Is to Increase awareness and develop professional kncwl-dge rnther than 
to serve as a basis for a decision about Implementation of a practice or program. 

Organizational Structure 

In cases vhcre the organizational structure of the Institution In which the 
client operates Is very diffuse and decentralized with decision making occurlng at 
many different levels, the problem of establishing benefit of information systems 
is Increased. In such cases. It Is difficult to even establish a concise definition 
of the client. Brown refers specifically to this problem when describing the 
evaluation of an Interunlverslty Information network.^*- On different occasions the 
client may represent an Individual, a committee, an organisation, or even an Insti- 
tution. With this sort of ambiguity In client definition, an evaluator may never 
obtain data about the outcomes or benefit of ar. Information system because of a 
simple failure to Identify the appropriate source to Investigate. 

Linkage of Means and Objectives 

In general, the questions of the relationship of means and ends and of the 
nature of causality arc extremely complex and much has been written about these 
Issues In philosophical literature. Certainly, these concepts are the subject of 
considerable debate and problems abound In their use. On the other hand, they are 
a definite part of common language and to Ignore consideration of the Issues they 
seem to imply In anj» social action program where there is competition for limited 
resources would seem to be inappropriate. 

Society seems willing to establish points in what may be a very apparent longer 
r.eans and ends chain as the designatcc goals of a given institution. These goals 
may not represent ultimate ends but they do mark a stag^ in an accepted causal chain 
leading to some more remote or ultimate end. In the case of educational institutions 
most of the designated goals have been defined in terms of behaviors which occ'ir in 
some point of time after formal schooling has ended. In addition to the designation 
of goals, society also seems willing to accept student attainment of certain skills 
rM knowledge as more immediate evid<3nca of the accomplishments of the schools. The 
causal links between knowledge of the organization of government in a high school 
civics class and adult civic responsibility seem to be accepted if only to make som« 
Ifivel of institutional accountability feasible. The degree of acceptance of a causal 
link is fluid and certain revisions are made ^vhen evidence from longitudinal studies 
become available but at an operational level definite causal links are accepted. 

The designers of various subsystems in educational Institutions must, in turn, 
either set objectives which are identical with those of the institutioa or, if tbete 
is some level of agreement about the existence of a causal link between the subsystem 
outputs and the institutional objectives, provision of these subsystem outputs can 
form the basis for evaluation. In either case, the evaluation of the subsystem is 
possible. However, when agreement about causal link is not present and the outputs 
of the subsystem are not identical with the institutional objectives, the evaluation 
process is such more difficult. This is a coinmon case for educational information 
systems. 



^^Brown, George W. An Interunlverslty Information Network? II. Evaluation. 
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In tooe cases, Information systems are designed around oUJe'-.rivcfi which are 
Identical to those of the client's Institution. This usually results In a disap- 
pointing assessment of the Information system because It does not have ooich of a 
chance of demonstrating substantial benefit. We simply lack the ability to be aulo 
to adequately distinguish the effects of separate responsible factors when these 
are not Independent. The best information system in the world will not appear very 
effective unless other responsible factors are also making their maxlmuai contribu- 
tions. For example, the main goal of the Pilot States Dissemination Program was 
to assist State Education Agencies to accelerate the improvement of educational 
practice and the instillation of tested innovations and programs by local school 
districts.*^ Such a goal cannot be attained without coordinated assistance from 
many components, other than the information system, in the school district. As 
stated by Paisley: 

We could argue that ERIC accounts for no measurable change In 
educational practice across the United States. The arguement 
is unfair: people, not information systems, "do" education." 

Another approach consists of defining information system objectives in terms 
of output and operational characteristics of the system. For example, in the list 
of nine objectives stated for MEDURS, which were used for several evaluation 
studies, none of the nine described client's institutional objectives. All were 
related to operational characteristics of the system. Also, when listing the "o^fe . 
specific objectives of ERIC both Paisley and Fry describe what seem to operational 
characteristics.^^ These include making documents accessible, preparing interpretive 
summaries, strengthen comownlcatlon c^^annels, develop a national information networl', 
and bring the ERIC knowledge base to the attention of practitioners. When this sort 
(>f approach is used, impressive objective accomplishment can be demonstrated but 
sunmative decisions related to funding and support are weakened by the lack of evi- 
dence of benefit. 

Evaluatio n Plan Assumptions 

Most of the difficulties described co far will become very evident to informa- 
tion system managers if they interact with an evaluator who is using an evaluation 
plan designed under the assumptions of a comprehensive/prescriptive decision model. 
A detailed description of both this and the incremental/remedial decision model, as 
these function in educational settings, has been provided by Stuff lebeam et. al. 
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Most descriptions of evaluation approaches seem to be based on an Implicit 
assumption that a comprehensive/prescriptive decision model Is appropriate. The 
evaluator will begin by requesting the information system managers to supply the 
objectives of the system and to identify the client group. However, to those who 
function in settings where document information is used for awareness purposes, 
by cliants who are difficult to identify, in an institution with diffuse decision 
making, concerning variables for which an accepted set of means and ends linkages 
does not exist, the fundamental organization of an approach represented by the 
evaluator, which is based on prior identification of ends, may be difficult to 
accept* Unless the evaluation plan more closely coincides with the assumption of 
an incremental/remedUl decision model which itself seems to be more suited to the 
context conditions described so far, the conflict over the ability to supply 
objectives and client identification as Inputs to the evaluation may result in both 
the evaluators and the system managers parting company with the conviction that the 
others are acting in a irrational fashion* 

Infoimatlon System Development 

Finally, the problems faced by an Information system evaluator are dependent 
upon the stage of development of the system during which the evaluation takes place. 
Alchot^h summatlve evaluation is undertaken to make a decision related to system 
acceptance, the past history of an Information system can never be disregarded* VRiett 
a pilot project is first begun the information system faces its most difficult period 
of evaluation. At this point a modern information system does not represent an 
incremental change in the schools. Since the system is new there are fewer organised 
aJvocacy groups to of£er support and to enter into the conflict resolution process 
raqulred in the incremental /remedial decision process* Concerns over benefits are 
high when a new system is being evaluated and less rime is available for resolutions 
of issues related to the identification of these benefits* 

Added to these problems is the increased emphasis on cost-benefit analysis 
caused in part by the general state of the econony* Cleverdon has identified 1970 
as a major turning point In the level of financial support available for Informntior 
systems and has Indicated the Impact of that change on evaluations* "Increasingly, 
those systems that are already established will have to Justify the costs that they 
Incur; proposals to establish new systems will be required to be subjected to a 
closer scrutiny*"*® 



SUGGESTED CHANGES IN EVALUATION FUNS 



In order to cope with the difficulties described, an evaluation approach is 
needed which is more capable of handling adverse context conditions* The following 
rccoamended changes in the design of summatlve evaluations for information systems 
may provide part of the answer* 
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Acceptanc e of Incompletely Stated Design Elements 



One change that Is reeded Is to develop an evaluation plan In which the Iderti- 
flcatlon of Infoxnatlon system objectives, measures of benefit, and target or client 
group Is accepted as part of the tasks to be accomplished during the evaluation. 
This Identification should be viewed as an Integral part of the evaluation l!:self 
rather than being demanded as Initial Inputs Into the evaluation plan. This chanpo 
of procedure may be required for the evaluation of other types of programs as well 
but It Is certainly required for Infornaton systcios because of our lack of the 
necessary knowledge. For example. In discussing what he calls "The Myths of Infor- 
tsatlon Needs", Ely describes our present state of knowledge of the Information needs 
of users as being Inadeguate for the purpose of specifying objectives for the design 
of Information systems.'^ 

The evaluation plan does not have to be completely open In Its Initial stages. 
There are some suggestions which can serve to Identify objectives, benefit, and 
client groups and those who have Initially proposed the Informatln system will have 
:-0Ke Ideas related to these matters. For example, Lancaster's list of suggested 
measures of benefit seem quite appropriate for most Information systems. 

1. Cost savings In using thl system as compared with costs of finding needed 
information elsewhere. 

2. Avoidance of loss of productivity (of, for example, engineers) that would 
result If Information sources were not readily available. 

3* Improved decision-making or reduction In the level of personnel required 
to make decisions. 

4. Avoidance of duplication or waste of cBglneerlng or research efforts on 
projects that have either been done before or that have proved Infeaslble 
by earlier Investigations. 

5. Stimulation of Invention (a serendipity factor) .^^ 

To this list other suggested measures of benefit could be added. For example, 
tlie amocnt a client would be willing to pay for infomation and the amount of time 
a person is willing to spend at information gathering activities, have been advocated 
by Herring et al.,^^ Vickery,^^ and Grotterer.^^ The evaluation of the Pilot States 
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project used feveral neasuref of benefit which seem very promiflng. Theee Included 
neaiures of the frequency with which cl'ents talk<^H to othera about retrieved Infor- 
mation, gave or shared copies of retrieved Information to others, or recommended 
the Information system services to others.** Wanger*s evaluation of ETxIC products 
also Included measures of the extent to which clients shared Information with 
others and, another possible measure of^beneflt, the extent to which clients were 
stimulated to seek further Information.^^ 

In the Pilot States evaluation there Is mention made of the lack of clear goals 
available at the beginning of the project. However, the solution advocated to this 
problem, both In the report Itself and In the position paper Included as Appendix I, 
does not seem adequate. The reconmendatlon made in the study is that consultations 
with program directors and staff along with deductive conclusions derived from an 
examination of literature about the program can be used to determine goals and 
objectives. These can also be put into some form of checklist for consideration by 
the system managers.^^ These suggestions are certainly valuable and were used well 
in Che report but they serve best only for inltiel stages of the avaluatlon. Also, 
the discussion of goal determination presented in the Pilot States report seems to 
place emphasis on the necessity of this type of work only in Che initial stages. At 
another point in the report the statement is made that an additional step in the 
Information transfer process, that of screening, was identified later in the project 
and would have been, if it had been anticipated, included in the objectives check- 
list.^^ What these conments seem to Indicate is the need for an evaluation design 
which uses all the methods described along with direct observation of the output and 
effects of the information ayatens and activltiea of dienta in order to formate goals 
and object Ivea throughout the evaluation process. This should be included in a formal 
way into the evaluation plan so that initial uncertainty about goala and object ivea 
13 not treated aa an underslrable wcakneaa in the progrco* 

Regardleas of how much initial information may be available, the evaluation plan 
should be set up so that data gathering activities are specifically included to 
obtain information about the abilities and resources which the information system 
actually creates in the client's institution. Data gathering activltiea will be 
needed to actually identify dienta and the uses which are made of information* 
Since theae activities cannot be expected to occur in the initial atagea of a ttudy 
this rort of approach say require more use of retrospective ceaaurea, which have 
maiqr weak points, but even a weak measure of an important outcome is better than 
r.o measure. 

In addition the evaluation plan should provide for activities designed to zathsr 
U&ta to decide whether or not it will be possible to link the observed system outputs 
and information uses of clients with any prespecifled objectives which moy have bwcn 
included as Initial input into the evaluation study. For example, some restrictions 
must be put on the form of objectives which are eventually set to guide a suonatlve 
decision to prevent these objectives from becoming identical with the overall insti- 
tutional objectives unless it is possible to separate out the effects of any other 
instftutlonal subsystem or input. Or, if these objectives are retained for some 
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political reason the evaluation plan will have to be modified to Include neasuree 
of the other relevant Input factors so that at least some sort of regression analysis 
tflll be possible. 

All of these considerations Involving observation, analysis, and refomulatlon 
of the evaluation design, will result In a lot of activity vhlch toore traditional 
evaluation plans presuce Is coii4>lcte before the actual evaluation stuuy begins. 
Ijhat Is required In th« approach advocated here Is nore of a willingness to accept 
incooplete elements of the plan initially and to provide actlvltlss to fill in more 
specific detail. 

Linking Formative and Suanative Variables 

After a reassessed and expanded set of objectives are available, the evaluation 
plan should provide for activities designed to gather inforaation needed to link 
the variables selected for formative purposes with the variables Implied by the 
system objectives* Earlier In this paper it was stated that the concerns of formative 
end suMative evaluation are distinct but related. They are related in that unless 
some evidence of linkage exists between the aspects of the system examined for forma* 
tive purposes and the variables chosen as indicators of output for suBaative purposes, 
the former would have to be dropped from the evaluation plan. Since objectives used 
for suiflmative purposes won't be formed at the initial stages of the evaluation, 
therm should be a stage specifically included where evidence will bo sought and 
usr;:^' to Judge the ertcnt to which satisfactory performance on formative variables 
ccucributes to attainment of the reformulated information system objectives. Once 
again, it is liecessary to consider an evaluation approach in vhlch there is tolerance 
for initial vagueness in defining variables for study. 

Lancaster has recoonended factors which seem suitable for an initial set of 
formative concerns so that the evaluatr.^n plan is not completely lacking in direction. 
These factors include coverage, re^^all, precision, response time, user effort, nnd 
form of the retrieved Infomation."^ Of these variables coverage and recall are not 
c:::a^ned very extensively in any of the four examples of evaluation reports but there 
is frequent mention of precision. The form of the retrieved information has also 
been a frequent concern with attention being directed to writing stylo and orjani^ta- 
tion by Wanger and the physical form of document reproductions with microfiche dis- 
cussed by Paisley.^' In future studies some of the listed factors may serve well 
b'jt it is possible that not all will be relevant due to some characteristic of 
educetion practitioners. For example, the role of response time is in some <iue8ti.>n» 
In the Pilot State evaluation it was reported that: 

...the main conclusion to be drawn from these data is that longer 
turnaround in itself does not reduce utilisation, « Ithough it 
might create dissatisfaction in the absence of a field agent. 

A variable which is often used in evaluations of educational information 
systems but vhlch is not included in the list previously mentioned is that of user 
knowledge of or familiarity with the information system. A logical argument is 
usually made to establish a link between use of the system, which is related to 
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benefit, and knowledac: of the systcin* The argument is presented that knowledge 
of the system is a losieal condition of necessity but not sufficiency with respect 
to use of the system. Paisley uses this variable as a basis for his analysis of 
ERIC and claims that smidies of knowledge of ERIC among educators show a failure to 
reach a wide audience.^^ Fry's study indicates a greater degree of knowledge of 
ERIC and Wanger's evaluation indicates 72 percent of a random sample report famil* 
larity with ERIC products .^^ The difference in the point of development of ERIC 
may explain these different results but with even the most favorable figures the 
strength of support offered for a sumr.?tive decision is reducad by the lack of 
evidence of relation between familiarity and some be:;' ^It related Imp'act. 

Identification ot Alternate Systems 

Once an accepted set of system objectives are available, the evaluation plan 
should provide activities designed to identify altamate information systems already 
in place in the client's Institution which have similar outputs as those described 
in the objectives list* Once true alternate systems are specified, then components 
nsust be identified. Finally, the evaluation plan must provide data about costs 
associated with these components. 

The reason for these activities seems clear, Suntmative evaluation or Justi- 
fication of contini'ed funding of a program is contingent upon being able to identify 
good programs. Since in general, school districts attempt to reach specified out- 
puts at minimum costs, desirable progrcms are one which offer greater benefit for 
;:imllar costs or the sar.e benefit at lower costs. All of these comparisons 
require the identification of alteruate systems. 

This requirement ;;ill probably be difficult to meet. In describing the results 
of survey? made of ppecialized information centers Kent has indicated that most 
vera not aware of cost Information for components of their own operations. Similar 
c-naents were made at a later date by Lancaster and Glllespie^-^ and Stephens.^^ 
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Of the four evaluations described In this paper only one mentions alternate 
systems and lu this one case there Is no attempt made to provide extensive comparl- 
tlve Information. In the Pilot State Dissemination Project evaluation the output 
of the Information system was considered comparable with that provided by consul- 
tants from the State Education Agency, In comparing the two the following conclu- 
sion was reached* 

In the first place, field agents can perform the job of help- 
ing clients at the local level better than consultants— as 
shown by our survey data,^'' 

No further Information was provided about cost for both of these systems or about 
the existence of nonccmparable outputs so no judgment about the most cost effective 
system to support could be reached in this case but such comparisons seem essential. 

It Is also possible that this activity will stir up hostility resulting from 
feelings of threat among Individuals associated with alternate systems. This may 
be unavoidable and certainly Is accepted and accomodated In the rationale of the 
Incremental/remedial decision model. However, If the evaluation plan does not have 
all of the features recommended in this paper the alternate system supporters may 
have a clear and unfair advantage over the supporters of the new Information system 
because of their greater organization. 

Provision for Dissemination of Evaluation Data 

To counter the advantage of organized groups associated with existlcg alternate 
system, the plan for the evaluation of a new information system should Include 
roeclfic activities for the dissemination of evaluation data. This process is essen- 
tial if there is to be a greater degree of consistency between the design of the 
evaluation study and the decision model of the client institution in cases where an 
incremental/remedial model is appropriate. The rationality of this decision model 
Is based on assumptions of conflict management involving negotiation and bargaining 
among members of informed self Interest groups. The formation of these Informed 
groups is dependent on systematic and comprehensive dissemination of evaluation 
information which must occur at all stages of the evaluation process. It U not 
endugh to prepare a final report which will be turned over to those who commissioned 
the study. Meitbers of the educational community who arc affected by the final 
suntnatlve decisions need to not only have access to evaluative data but they need 
to be alerted to the existence of this data. They need to be as aware of the eval- 
uations as they are of the information systems. 



CONCLUSIONS 



Sunmative evaluation Is not often popular because of the final nature of the 
activity. Perhaps educators are more reluctant than other groups to accord sunma- 
tive evaluation a respected status since so much of the literature in the field of 
education has stressed the importance of the development of individual potential 
and a high level of dedication to finding ways to reach students or improve prog:a:)9. 
All of the examples reported in this paper have been extensively directed to format iv 
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evaluation even though program funding decisions have been frequent. WUen con- 
sidering the fact that such decisidns have been made it is very difficult to under- 
stflVid the almost complete lack of a comparative evaluation of alternate informatim 
systems vhich include some of the newer and some of the more established methods of 
serviftg the needs of educational practitioners. This. lack of data becomes very 
apparent when resources are sought to establish an information system* 

Competition for scarce resources will probably always be an element affecting 
program design and evaluation in education. The interests of no group will be 
served well if poor decisions are made and resources are spent on ineffective pro- 
grams. However, at present it is very doubtful whether or not an innovative infor- 
mation system designed for educators has much of a chance competing for funds even 
if i*- is effective because of the way evaluations arc traditionally structured. 
Some accomodations have to be made in the evaluation design and more conscious 
attention must be given to formal sumraative evaluation if this situation is to 
change. The recommended changes presented in this paper may be a step in that direc- 
tion. 



